This invention relates generally to the field of computer input and more particularly to a process for increasing the speed and accuracy of computer input for persons who have difficulty using, cannot use, or choose not to use a standard computer pointing device (e.g., mouse) or character input device (e.g., keyboard). The invention is further related to the field of assistive technology as a process for improving physically disabled persons' access to computers as well as improving their speed and accuracy. The invention is further related to modifying existing switches or creating new switches which provide a mechanism for converting a non-dichotomous user response (e.g., pressure; muscle activity; voice volume; distance moved; electrophysiological action such as that measured by electromyography, electro-oculography, or electroencephalography, etc.; proximity to a sensor; point of gaze; etc.) from a standard dichotomous output (on/off) to a digital or analog monotonically increasing or decreasing output. The invention also is related to an apparatus that assists users who have adequate control in one direction of one dimension (e.g., moving a mouse to the left) but who have poor control of the antagonistic muscle groups performing movement in the opposite direction (e.g., moving the mouse to the right), by providing a counterforce to augment movement in the opposite direction. The invention is further related to providing a diagnostic tool to help in a clinical or research setting to determine which user response is the best candidate, which switch/controller is the best candidate, which combination of response and switch/controller is best, etc.
Although not limited to pointer input device emulation and text entry, the simplest form of the invention is most easily understood in the context of standard computer input and control such as using a mouse pointer to point and click on an option presented on a computer display or typing to a word processing program. Consequently, the following general discussion uses the examples of pointing and clicking on a screen location and typing words into a word processing application as the general context in which the invention is described. Also, although not limited to use by the physically disabled community (e.g., a soldier or driver who cannot divert gaze to a standard two-dimensional screen and perform conventional selection tasks could benefit from this invention), the following discussion uses the general example of increasing computer access for a physically-disabled computer user as the general context because the use of the invention is more easily understood in that setting and the potential advantages are more obvious.
The invention described here builds on current knowledge of general computer input technology. Six specific such technologies are described here because of their relevance for the invention: pointer-input technology, mouse-emulation technology, on-screen keyboard technology, “scanning” keyboard technology, list-based typing, and control device technology.
Background on Pointer-Input Technology. Over the past 20 years, the use of a “mouse” pointer as an input device for a computer has become so ubiquitous that many people would assume it has always been one of the primary input devices. Although the best known, the mouse is just one of a family of many possible “pointers” that can be used with today's computer. Other more well-known members of this family include the trackball, joystick, touch screen, touch pad, and graphics tablet. For all such pointers, the basic underlying response requirements for a user usually are the same: 1) identify a specific location on a two-dimensional computer display (i.e., “point” by positioning the mouse cursor on the desired target), and 2) signal a selection or other instruction (e.g., click, double click, drag, etc.).
For a variety of reasons, more conventional pointers might not work for some users or for any user in some situations. The most intuitive example is a physically disabled person who does not have the hand/arm dexterity to use a conventional pointer. As a result, innovative new pointers have emerged which attempt to exploit the capabilities of such users. For example, sophisticated (and expensive) pointer systems based on head movements and point of gaze are commercially available. There even have been reports of successful attempts to use measures of brain activity as a mechanism for positioning a mouse cursor. Another existing and less expensive technological approach is based on identifying a single response that the user can perform reliably and using that single dichotomous switch action to direct the location (e.g., point) and select (e.g., click), usually accomplished by using the mouse. This general approach has been called “mouse emulation” and the corresponding hardware/software approaches used are called “mouse emulators.”
Background on Mouse-Emulation Technology. Although different approaches can be taken for emulating a mouse, the discussion here is limited to an approach that is commonly used by persons with physical disabilities called “scanning” or “switch scanning.” Even though a person might lack the ability to precisely use a conventional mouse, a single response often can be identified which the user can perform reliably (e.g., blink an eye, move the thumb, puff into a straw, etc.). In single-switch scanning approaches, the user typically performs such a response to signal selection of an option that has been presented by the computer. While there are several commercial products available, the example used here is an existing product of Applied Human Factors, Inc. (AHF) called “ScanBuddy,” which is representative of such products.
Mouse emulation programs allow a user to perform most mouse functions with a single switch. As with any pointer, the two major tasks are to identify a location on the display and then instruct the computer what to do at that site. To identify the point, the user initially operates the switch to begin the program (depending on the adaptation, operating the switch might mean blinking an eye, puffing on a tube, touching a contact switch, etc.). At that time a rectangle is drawn surrounding the top half of the display. If the target point is inside that rectangle, the user clicks the switch; if it is not, the user waits for the (adjustable) time period to pass at which time a rectangle is drawn around the bottom half of the display. Once the correct section of the display is selected, then the selected rectangle is divided in half and the two new sections offered to the user one at a time. In the current example, if the user initially selects the bottom half of the display, then the left half of that section (quarter) is offered. Again, if the target is in that rectangle, the user clicks the switch and if not, the user waits for the other (right) half to be offered. Alternatively, if the mouse emulation software detects that the user last clicked inside a dialog box, it can offer the dialog box first before offering the entire display (as shown in Frame 1 of FIG. 1). AHF calls this method “section scanning,” and this is the only product that uses this method (as far as AHF can determine). It is intended to provide a fairly quick way for a user to identify the general location (section) of the display that contains the target. AHF determined that, while this method is useful to rule out a large portion of the display, it is not efficient as a way of selecting the target point (in fact, while users can adjust how many section divisions are offered, the mouse emulation software limits the number to five or less).
Once the section has been determined, the scanning method changes to “line scanning,” a method that is used in other existing products (for the entire display—not for a selected section). Specifically, a horizontal line is presented at the top of the final section and moves downward at an adjustable speed. The user's task is to operate the switch when it crosses the targeted point (hence, establishing the vertical location of the target point). Then, an icon showing a pointing hand is moved horizontally along that line from left to right, and the user's task is to click the switch when the finger points to the target location. See Frame 2 of FIG. 1 for graphic representations of vertical (top) and horizontal (bottom) line scanning. Because the speed of the scanning lines can be set to complement the ability of the user, line scanning provides considerable accuracy when selecting a point on the display. However, line scanning alone can be very slow if the target is in the opposite corner from where the lines begin. Hence, this combination of section and line scanning was introduced as a way of increasing speed while retaining accuracy.
At this point, the user has accomplished the first task (point) and now must provide the instruction about what action should be taken at this point (click, double click, etc.). Here, ScanBuddy again is representative of existing products, offering a scanning menu containing click, double click, drag, cancel, and special (see Frame 3 of FIG. 1). If the “special” option is selected, another scanning menu appears containing right (right mouse button), scroll, shift+ (e.g., hold the shift key down while dragging), control+ (e.g., hold the control key down while clicking), pulse click, favorite (launch a selected application), and settings. Hence, most of the mouse functions that can be performed with a regular mouse can be performed by a person using a single switch with a mouse emulator. The problem is speed, no matter how clever a mouse emulation program is, it is hard to imagine that one could ever approach the speed of a non-disabled person using a standard pointer.
Background for On-Screen Keyboard Technology. Traditionally, the hardware keyboard has been the standard input mechanism for typing when using a computer. However, many people (e.g., persons with disabilities, children, new typists, or infrequent typists) can experience difficulty using hardware keyboards for a variety of reasons such as poor dexterity or unfamiliarity with the keyboard layout. One existing technology that can be useful is the on-screen keyboard. In this approach users are provided a graphic representation of a keyboard on the computer's display. Using a “point-and-click” input mode, a mouse or other pointer is used to type letters by clicking on the graphically displayed keys or by touching them with a finger or a stylus when a touch screen is available. When selected, the appropriate character typically is sent to a word processor, spread sheet, database, or some other open application. On-screen keyboards can be useful because the keys can be presented in a more meaningful way (e.g., an alphabetic layout instead of the traditional “QWERTY” layout), which can be easier to use for some people. Also, with an on-screen keyboard the names of the letters can be spoken (with synthesized or recorded speech) when they are selected which also can be useful for some people. In addition, pictures can be added to a key's face, and this can be helpful (e.g., a picture of an “apple” on the “A” key). Finally, useful information can be presented when a key is “pointed at” with the pointer. For example, when a user points at the key with the label “A,” the key's appearance can change; a picture of an “apple” might appear, a new label “A as in Apple” might appear and be read aloud. All of these features can be especially useful to a person with spelling, reading, or writing learning difficulties. FIG. 2 shows representative examples of a traditional “QWERTY” keyboard in Frame 1 and an example of an “Alphabetic” keyboard in Frame 2. Some people are unable to use a traditional hardware keyboard because of a physical disability. If they are able to use a pointer, then on-screen keyboards provide a very important access tool. Similarly, some persons have a more severe physical disability and for them, other strategies must be applied such as “switch scanning” techniques.
Background on “Scanning” Keyboard Technology. The third technology deals with “switch scanning input methods”. While the point-and-click input mode is the best approach for many users, some people cannot use a standard keyboard or a standard pointing device. However, for most such individuals, some reliable single response usually can be identified and used to make a selection with a switch. Switches are available to capture a variety of responses more likely to be under the user's control such as the eye blink response, air puff or air sip response, etc. The single switch then can be used to select options being offered on the display such as keys being “offered” on an on-screen keyboard. Such “scanning keyboards” allow the output from a simple switch to signal the selection of a group of keys or individual key being offered.
In one existing method called single-switch scanning, an on-screen keyboard offers different options at a constant rate, for example, every 2 sec. Typically, the user first selects the “row” that contains the target letter and then selects the “column” or key, hence, this approach is called “Row/Column Scanning.” This method is illustrated in FIG. 3. In Frame 1, the top row of keys (i.e., containing Tab, Qu, W, E, R, etc.) is first “highlighted” (e.g., its color changed, a box drawn around it, etc.), for the pre-established time period. Next, the second row is offered to the user, again by highlighting, as shown in Frame 2. In this example, the user is typing the letter “A,” so the second row is selected by the user through the use of the switch. Next, the columns (i.e., keys) in that row are systematically highlighted from left to right—as shown in Frames 3 and 4. The user selects the correct key by operating the switch when the target key is highlighted, in this example, when the “A” key is highlighted as in Frame 4. At that point, the key is typed and the character is sent to the open software application. Scanning then is reset so the user can type the next letter. Although tediously slow, scanning is the only typing alternative for many users.
Background on List-Based Typing. Recently, innovative new procedures for increasing typing speed and accuracy have been created. One that is especially relevant for the present invention is labeled “list-based typing,” and was submitted as a patent application on Oct. 15, 2002 (A process for integrating visual, cognitive, and motor tasks during typing and character string entry: jc930 U.S. Ser. No. 10/271,419) and is still being evaluated at the time this application was submitted. When using list-based typing, the first letter is selected from an on-screen keyboard (as shown in Frame 1 of FIG. 4). In this example, the user is typing the word “your.” When the initial “y” key is selected (either by pointing and clicking or by scanning), that key opens up showing a list of options (as shown in Frame 2). Each option contains the current “Root” (the letter(s) already inputted) and a candidate “Extension” (letter or letters known to follow the Root based on words in the dictionaries currently opened). The user's task is to select the option that is most consistent with the beginning of the word being typed, so in this case, the “Yo . . . ” option is selected, resulting in another list of options as shown in Frame 3. Selection of the “You . . . ” option in Frame 3 produces the list shown in Frame 4, which contains the target word “Your.” There are several advantages to this approach, but the most significant benefits are for persons using scanning. For example, only one switch operation is required per selection (compared with two for traditional row/column scanning), so “work” is cut in half once a list is introduced. Also, speed is increased because the options are ordered by their probability (as shown in FIG. 4). However, options also can be ordered alphabetically, which is perhaps more appropriate for persons using point-and-click input. The most important aspect of List-Based Typing for the present invention is that options are presented in a one-dimensional list format.
Background on Control Device Technology. Control devices can be characterized by the user's behavior required to operate the device (Input); the nature, form, and function of the resulting output (Output), and the internal mechanism(s) that sense and transduce the user's behavior to the final output (Sensor/Transducer). For the purpose of this discussion, two general categories of computer interface input controls are identified to provide a frame of reference and a third type is offered/described because it is as an important component in the invention.
Pointers compose the first general category of control device to be discussed here. As introduced above, there are a variety of pointers available ranging from the standard mouse to more complex eye-tracking systems. As a group they vary widely with regard to the Input required from the user as well as the Sensor/Transducer utilized to characterize the user's Input. However, with regard to functional Output they all are very similar, that is, they result in the movement of a mouse cursor in two dimensions independently on a computer's two-dimensional display.
Switches compose the second general category of control device discussed here. Dozens of simple switches with dichotomous output are incorporated on every computer hardware keyboard. Also, as discussed above, switches with dichotomous Output are exploited in assistive technology in “scanning” approaches used for emulating a pointer (see the mouse-emulation section above) and for inputting text or other keyboard commands (see the scanning-keyboard section above). As with pointers, their Input might vary significantly (e.g., contact pressure, proximity, movement, a vacuum, air pressure, sound, electrophysiological activity, etc.). Also similar to pointers, the Sensor/Transducers can vary widely depending on the Input characteristics. However, as with pointers, they all have a similar (but different from pointers) Output characteristic: a dichotomous on/off state that usually is then interpreted by the computer based on hardware or software “instructions” as a “selection” of a computer action.
The invention described here utilizes a third type of control device as the critical component. This type of control certainly is not new or novel and significantly predates pointer controls. While providing less control information than conventional pointers, it is capable of generating significantly more control information than a conventional switch. Specifically, the control device relevant for this invention allows user Input in one dimension rather than two. In such a control, some measurable dimension of user Input is sensed/transduced to some monotonically related Output measure. Common examples include a gas pedal, brake pedal, steering wheel, dimmer switch for an overhead light, etc. Indeed, a computer pointer can be conceptualized as two integrated orthogonal one-dimensional controls (although pointers actually are more complex because they meaningfully consolidate simultaneous Input from two independent Input dimensions). Such switches sometimes are labeled as proportional controls in the literature.
Three important points should be made regarding the relationship between the invention and one-dimensional controls. First, while one-dimensional controls are very common in general, they are very uncommon as commercial computer controls. The exceptions are custom game interfaces in which, for example, a steering wheel or gas pedal are parts of a game or simulation. Second, through software modification, an ordinarily two-dimensional pointer can be modified so that the resultant control is one-dimensional, and while that doesn't usually seem like a useful approach, it is an important characteristic that is exploited with this invention. Third, while many existing custom switches designed for assistive technology applications have been created in a way that the eventual Output is a dichotomous switch closure (in order to allow a physically-disabled person use a scanning approach—as described above), many could be modified to be one-dimensional controls (i.e., by exploiting the underlying non-dichotomous output from the transducer before it is converted to a dichotomous Output). Part of the invention is an apparatus that captures the more basic output from existing sensor products and transforms it into an output that is compatible with the input standards and conventions required for standard computer pointers.
Statement of the Problem: There is a Need to Help Persons with Mobility Disabilities to Interact Faster and More Accurately with Computers without Having to Rely on Very Slow Single-Switch Scanning Techniques.
The transition from card, tape, and keyboard-based computer input/control mode to a two-dimensional (2-D) graphic user interface control mode (e.g., the mouse, joystick, trackball, etc.), was one of the most significant innovations in the evolution of computer interface technology. However, the change from keyboard-based “control keys” to tasks involving locating a cursor by controlling its location in two independent dimensions and then using a switch to signal selection (i.e., pointing and clicking) was not a very useable improvement for persons with physical disabilities. Within the field of assistive technology, the general strategies adopted were to attempt to help the physically-disabled user adapt to conventional 2-D pointers, create custom 2-D pointers based on some behavior more likely to be controllable by such users (e.g., eye-gaze systems), or emulate 2-D pointers using switches combined with scanning techniques, which are extremely slow, tedious, and boring alternatives. Clearly there is a need for improved methods/technology to make computer access (i.e., mouse emulation and character input) faster and less tedious for persons with physical disabilities.
When the primary control mechanism for personal computers shifted in one step to an on-screen, two-dimensional control strategy, it is possible that a potentially useful control strategy was literally skipped over. Specifically, it is likely that subpopulations of users exist who have adequate control in one dimension but who do not have adequate control in two independent dimensions simultaneously. There is evidence in the literature to support this assertion, that is, that some people might benefit by operating a one-dimensional control devised in a way that emulates a standard computer pointer and a hardware keyboard.
Methods are needed which allow a person to have improved computer access, input, and control by using a one-dimensional control device. The “one-dimensional interface” process described herein is such a process in which the user operates a control mechanism based on changes in behavior measured in one dimension. Also, such a one-dimensional approach can be modified to further exploit the capability of a specific user by providing two-directional control along that single dimension found to produce the most accurate control for that person (e.g., a specific user might have more control when moving an arm left and right compared to up and down). Hence, in this application the term “One-Dimensional Control” is used to represent the general approach of limiting control to one dimension and the term “Two-Directional Control” is a special case in which changes in both directions impact corresponding changes on the computer's display.
In addition, there are studies in the literature showing that some individuals have more control in one direction along a single dimension than in the opposite direction (e.g., a person might be able to move an arm to the left faster and more accurately than moving it to the right). This is not surprising because antagonistic muscle groups are involved and control of a muscle group that results in more controlled movement in one direction does not necessarily mean that the antagonistic muscle group is equally well controlled. Hence, another form of the invention is labeled a “One-Directional Control” because in that method, changes in only one direction of a single dimension are used to produce corresponding changes on the computer's display. Hence, for the person described above, only arm movement to the left is used to determine the cursor's position (when emulating a mouse) or character selection (when emulating keyboard input). In addition, for such users there is a need for an apparatus to augment the performance of the antagonistic muscle groups, in this example the apparatus provides a counterforce to help the user return the arm to the starting position. Also, because of the diverse preferences among users with regard to what approach is the most useful for them, different specific methods and strategies are needed for both mouse and keyboard emulation. Methods also are needed to allow such an interface to accommodate input from standard existing pointer controls as well as new methods to modify existing controls and create new controls to capture a user's behavioral responses in one dimension. Finally, methods are needed to assess the usefulness of both Two-Directional and One-Directional control methods relative to other possible methods, as well as assess different candidate user behaviors, control devices, settings (speed, direction, gain, etc.), interface strategies, etc., with the goal of selecting or recommending the best input strategy and maximizing that user's performance.